This PR adds the ability to write local CloudFormation modules, which are YAML files that contain Parameters and Resources. These modules can be included in a parent template in a new Modules section. Modules are configured by a combination of formal Parameters (which coincide with the Properties defined in the parent) and an Overrides attribute, which allows users full control over the rendered output. This is a port from the functionality that exists in the Rain CLI. See the snippets below for a basic example.

Parent template

Modules:
  Content:
    Source: ./basic-module.yaml
    Properties:
      Name: foo
    Overrides:
      Bucket:
        Properties:
          OverrideMe: def

basic-module.yaml

Parameters:
  Name:
    Type: String
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref Name
      OverrideMe: abc

Output from the package command

Resources:
  ContentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: foo
      OverrideMe: def

This PR is a proof of concept with a limited set of functionality.

Design Decisions

1. Modules can be imported into a template (or into a module itself) from a new Modules section. In Rain and in registry modules, this section does not exist. Instead, the imported content is configured in a resource and the source of the module is embedded in the Type attribute. This PR implements both ways of configuring an imported module, giving customers the option of whichever feels more intuitive. The benefit of using a Modules section is that it will be more intuitive if we decide to allow injecting other sections other than Resources, like Outputs.

2. Logical IDs are a concatenation of the module name and the logical id defined in the module. Unlike registry modules, users do not have the option of referring to the resources with a dot between the names.

3. The Overrides section is not considered an escape hatch of last resort. It is an encouraged way to configure modules without requiring the module author to anticipate every possible need. IaC abstractions are leaky by nature and we expect users to understand the internals of any module they import. Deciding how many actual Parameters to include in a module is highly opinionated and left up to authors. One caveat here is that Overrides are fragile, since they will break if the module author changes any of the logical IDs in the module. See item 5, Outputs, as a way to mitigate this.

4. This functionality has been added to the aws cloudformation package command. We could have created a new dedicated command for module packaging, like synth. This is an easily reversible decision. The code is ported from the Rain pkg command, which was originally written to mimic the behavior of aws cloudformation package, so it seems like the right place to put it. One annoyance is that if you are only doing local packaging, you still have to supply the s3-bucket argument, and you need to be connected to an AWS account.

5. Modules can define Outputs, which are key-value pairs that can be referenced by the parent. This gives module authors a formal way to pass values to the consumer without them needing to rely on Overrides, which will be fragile if the module author changes any of the names in the module.

6. Instead of using Fn::ForEach as a way of implementing loops, we added a Map attribute to a local module resource. The module output is repeated for each element in the map, substituting $MapValue and $MapIndex in module Properties. This can only be used with lists that can be resolved at build time.

What's not done in this PR

We need to decide what's necessary for an initial version and what can wait

[ ] - Many of the related Rain features like Metadata commands and Constants
[ ] - Conditionals in modules, to emit or omit resources based on Parameters

How to test this PR

Check out my branch and install the aws cli into a local Python environment.

git clone [email protected]:ericzbeard/aws-cli.git
git fetch origin cfn-modules
git checkout cfn-modules
python3 -m venv .env # Or whatever you use for python environments
source .env/bin/activate
pip install -e .
aws cloudformation package --s3-bucket [A_BUCKET_YOU_OWN] \
    --template-file tests/unit/customizations/cloudformation/modules/type-template.yaml